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Abstract 

Psychological investigations have led to considerable in- 
sight into the working of the human language compre- 
hension system. In this article, we look at a set of prin- 
ciples derived from psychological findings to argue for 
a particular organization of linguistic knowledge along 
with a particular processing strategy and present a com- 
putational model of sentence processing based on those 
principles. Many studies have shown that human sen- 
tence comprehension is an incremental and interactive 
process in which semantic and other higher-level infor- 
mation interacts with syntactic information to make in- 
formed commitments as early as possible at a local am- 
biguity. Early commitments may be made by using top- 
down guidance from knowledge of different types, each 
of which must be applicable independently of others. 
Further evidence from studies of error recovery and de- 
layed decisions points toward an arbitration mechanism 
for combining syntactic and semantic information in re- 
solving ambiguities. In order to account for all of the 
above, we propose that all types of linguistic knowledge 
must be represented in a common form but must be 
separable so that they can be applied independently of 
each other and integrated at processing time by the ar- 
bitrator. We present such a uniform representation and 
a computational model called COMPERE based on the 
representation and the processing strategy. 

Introduction 

Psychological investigations of human language process- 
ing have led to considerable insight into the working of 
the language processor. Yet, attempts to build psycho- 
logically real computational models of language process- 
ing face considerable problems in translating the con- 
straints put forth by psycholinguistic theories into the 
representations and processes that constitute the com- 
putational model. In this article, we summarize a large 
body of evidence from psycholinguistic literature into a 
set of principles and use them to derive a computational 
model of human sentence processing that employs a par- 
ticular control of processing and a uniform representa- 
tion of all linguistic knowledge. 

We argue that, in order to explain a large variety of 
findings in human sentence comprehension, the compu- 
tational model must employ a single arbitration pro- 
cess that can integrate independently proposed syntac- 
tic and semantic information to resolve ambiguities. We 



present a uniform representation for syntactic and se- 
mantic knowledge that enables syntactic and seman- 
tic interpretations to be integrated through intermedi- 
ate representations. The representational primitive is 
a node that specifies part-of and has-part relations to 
other nodes, preconditions on these relations, and ex- 
pectations that could be generated from such relations. 
We show how such representations could be employed 
by a parser that makes the right commitments at the 
appropriate times in order to explain a variety of hu- 
man sentence processing behaviors such as incremental 
interpretation, immediacy of semantic and conceptual 
interaction, modular behaviors such as purely structural 
preferences, early commitment and error recovery, gar- 
den paths, and resource-limited delayed decisions. We 
have developed a model called COMPERE (Cognitive 
Model of Parsing and Error Recovery) based on such an 
architecture and tested it in a computer implementation. 

Psychological Constraints on Language 
Comprehension Models 

Experimental observations of human sentence processing 
behavior can be summarized in terms of a set of princi- 
ples including the following: 

1. Incremental Comprehension: 

Psychological experiments have confirmed the general 
intuition that human language comprehension is an in- 
cremental process of progressively building a syntactic, 
semantic, and referential interpretation of a sentence, 
(e.g.. Grain and Steedman, 1985; Marslen- Wilson and 
Tyler, 1987; Steedman, 1989; Taraban and McClelland, 
1988). 

2. Early Commitment: 

The sentence processor must make early commitments 
as soon as it has the information necessary to do so 
(e.g.. Grain and Steedman, 1985; Frazier, 1987). Re- 
source limitations such as working memory capacity and 
the real-time nature of the comprehension task forbid 
the processor from pursuing all possible paths in parallel 
until they are ruled out (MacDonald, Just, and Carpen- 
ter, 1992). It must use information available from every 
knowledge source to make a choice as early as possible. 



3. Delayed Decisions: 

However, the processor must pursue parallel interpreta- 
tions sometimes to account for delayed decisions. When 
syntactic and semantic preferences are in conflict, the 
processor must postpone the decision until further infor- 
mation becomes available (Holmes et al., 1989) or un- 
til a decision is forced by limits of memory resources. 
Moreover, at the resource limit, it appears that syntac- 
tic preferences override any semantic preferences (Stowe, 
1991). Delayed decisions have also been observed in the 
presence of lexical ambiguities. 

4. Error Recovery: 

Early commitment inevitably leads to erroneous deci- 
sions at times when later information proves an earlier 
choice incorrect. The processor must also be able to 
recover from such errors by switching to an alternative 
interpretation (e.g., Eisclt, 1989; Stowe, 1991). This er- 
ror recovery whether in structural or lexical ambiguity 
resolution, should be possible through a repair process 
that is incremental. Some parts of the erroneous struc- 
tures should be reused and others repaired to result in 
correct structures. Mere reprocessing from scratch does 
not explain the grades of difficulty exhibited by weak 
and strong garden-path sentences (Abney, 1989). 

5. Interaction: 

Apart from incremental processing, the processor sup- 
ports interaction between different faculties such as syn- 
tax, semantics, and reference (e.g., Grain and Steedman, 
1985; Marslen- Wilson and Tyler, 1987; Taraban and Mc- 
Clelland, 1988). Natural languages are replete with am- 
biguities which cannot all be resolved by the use of any 
one kind of knowledge. An incremental processor builds 
interpretations of an incomplete sentence at all the dif- 
ferent levels that interact with each other and chooses an 
interpretation that is best with respect to all the knowl- 
edge available to the processor at that point. Incremen- 
tality, interaction, early commitment, and error recovery 
can together explain the ability of the human language 
processor to deal with the proliferation of local ambi- 
guities in natural languages (e.g.. Grain and Steedman, 
1985). 

6. Functional Independence: 

A final constraint derivable from a vast body of psy- 
cholinguistic studies can be summarized by stating that 
each kind of knowledge must be capable of being applied 
independently of other kinds. For instance, it must be 
possible to apply purely structural principles to resolve 
an attachment ambiguity for an adjunct. This syntactic 
knowledge must be represented so that it can be ap- 
plied no matter what lexical items are involved in the 
adjunct (Prazier, 1987; Frazier, 1989). Independence is 
also necessary to explain the behavior seen when de- 
lays in commitment are terminated by resource limits 



(Stowe, 1991). Another important source of evidence 
for functional independence between levels of language 
processing lies in behavioral studies with aphasic sub- 
jects (e.g., Caramazza and Berndt, 1978). Though the 
neurological independence between syntactic and seman- 
tic knowledge has been questioned by recent studies of 
cross-linguistic aphasia (Bates, Wulfeck, and MacWhin- 
ney, 1991), functional independence would still be nec- 
essary to explain the diff'erential impairment of syntactic 
and semantic abilities in aphasia (Mahesh, 1993). 

Moreover, in order to make sure that the actual course 
of steps in the model agrees with human behavior, not 
just the end results of processing, the representations, 
processes, and resources used by the model must all be 
truthful to psychological data. 

The need to combine bottom-up and 
top-down processes 

Data-driven models of language comprehension with 
bottom-up strategies are compatible with incremental 
semantic interpretation. Incremental comprehension is 
best described by a bottom-up strategy that interprets 
each successively larger constituent as it is built from the 
next word in the sentence. A top-down strategy would 
force the processor to commit to a whole constituent be- 
fore; analyzing the parts of the whole. Making such com- 
mitments when the processor does not have necessary 
information results in unwarranted backtracking. 

Standard theories of syntax using phrase-structure 
rules are incompatible with incremental processing given 
that the steps taken by the parser be psychologically 
real (Steedman, 1989). Though it is possible to apply a 
bottom-up strategy to phrase-structure rules, the result- 
ing process will not be incremental since the bottom-up 
parser waits until it has seen every daughter of a con- 
stituent before interpreting it. Phrase-structure rules 
can be applied to carry out an incremental interpretation 
only if sentences have a left-branching structure which 
is not true with a majority of natural languages (Steed- 
man, 1989). What we need is a bottom- up parser with 
top-down guidance that can make early commitments 
before actually seeing every part of a constituent so that 
semantic interpretation can be incremental. Such early 
commitments may be made by employing top-down in- 
fluence from a variety of types of knowledge. 

Types of Top-Down Guidance to 
Bottom-Up Parsing 

Information providing top-down guidance to the pro- 
cessor can be of three types: grammatical information 
about the categories involved (Steedman, 1989), general 
structural principles (Frazier, 1989), and feedback from 
semantics, reference and discourse interpretation (e.g., 
Grain and Steedman, 1985; Taraban and McGlelland, 
1988). 



1. Syntactic Expectation: 

Grammatical information tells the processor about the 
arguments that must follow before the current con- 
stituent can be complete (and hence grammatical). For 
instance, after seeing a noun phrase, a verb must follow 
for the sentence to be complete. We can say that the pro- 
cessor can expect to see a verb phrase at this point. The 
bottom-up parser can use such syntactic expectations to 
make early commitments at syntactic ambiguities. This 
grammatical preference for expected structures results 
in the same behavior as expected by the minimal at- 
tachment principle and explains garden-path behavior 
in reduced relative clause sentences such as (1). 

(1) The officers taught at the academy were 
very demanding. 

2. Semantic and Pragmatic Preference: 

One way to introduce extra-grammatical top-down influ- 
ence on attachment ambiguities is through interaction. 
Semantic and discourse processes feedback to syntactic 
processing and exert preferences for some attachments 
over others (e.g.. Grain and Steedman, 1985; Stowe, 
1991). For instance, semantic feedback can tell the pro- 
cessor that the prepositional phrase (PP) in sentence (2) 
must be attached to the noun phrase (NP) since a horse 
cannot be used as an instrument for seeing. 

(2) The officer saw the soldier with the horse. 

3. Structural Preference: 

The processor must also be able to exert purely struc- 
tural preferences that are independent of the categories 
and lexical items involved in the ambiguous parts of the 
sentence. Examples of such a preference are right as- 
sociation and minimal attachment (Frazier, 1987). Such 
syntactic generalizations allow the syntactic processor to 
make early commitments for adjuncts as well, thereby 
explaining several syntactic phenomena (Frazier, 1989). 
There is psychological evidence in structural ambiguity 
resolution which demonstrates the need for this kind 
of top-down influence. For instance, Stowe (1991) has 
shown that the human sentence processor delays a deci- 
sion (i.e., does not do an early commitment) when there 
is a conflict between syntactic and semantic preferences 
at a structural ambiguity (i.e., when top-down influence 
from grammar or structural principles and those from 
semantic feedback contradict each other). Experiments 
showed that people continue to delay the decision and 
pursue multiple interpretations until they reach resource 
limits. At the limit, they make a choice based only on 
syntactic preferences. This result would be left unex- 
plained but for the presence of a top-down influence of 
the structural kind. 



The Control of Parsing 

Top-down guidance from syntactic expectations and 
structural preferences can be integrated with a bottom- 
up control of parsing by employing a form of left-corner 
parsing (Abney and Johnson, 1991). We have developed 
a variant of left-corner parsing by adding to it the virtues 
of head-driven parsing. The resulting mechanism that 
we call Head-Signaled Left Corner Parsing produces the 
right sequence of syntactic commitments to account for 
a variety of data. The parser has been implemented in 
the GOMPERE model. Lack of space prevents us from 
further describing the parsing mechanism here, but it is 
described elsewhere (Mahesh, 1994). 

The Need for an Arbitration Mechanism 

While top-down and bottom-up influences on sentence 
comprehension arising from syntactic sources of knowl- 
edge can be combined in the parsing algorithm men- 
tioned above, the sentence processor must also combine 
information coming from semantic and conceptual analy- 
ses with the syntactic preferences. This communication 
between syntax and semantics is necessary to account 
for evidence from interaction, delayed decisions, early 
commitment, and error recovery. The left-corner pars- 
ing algorithm merely identifies the points when the com- 
munication ought to occur but doesn't tell us how the 
communication is handled or how conflicts are resolved 
in the best interests of the constraints on behavior. One 
way to do this is to integrate the representations of the 
two types of knowledge a priori as in semantic grammars 
and what are called grammatical constructions in certain 
other models (e.g., Jurafsky, 1991). Such an approach 
suffers from reduced generativity and other disadvan- 
tages (see the Related Work section below). In order 
to avoid losing generativity, the sentence processor must 
keep the knowledge sources separate and introduce an 
arbitration mechanism that dynamically combines infor- 
mation arising from independent syntactic and semantic 
sources, resolving any conflicts that might arise in the 
process. 

The arbitrator needs to combine information coming 
from independent syntactic and semantic sources which 
talk in different terms. Syntax describes its interpreta- 
tions in terms of grammatical relations such as subject 
and object relations while semantics talks in terms of 
thematic roles.|^ The arbitrator must establish corre- 
spondences between these two representations (and be- 
tween the decisions made by the two) . One way to bridge 
this communication gap between the languages of syn- 
tax and semantics is to add a translation procedure in 

^For purposes of illustration and simplicity, we are em- 
ploying thematic-role assignment using selectional prefer- 
ences from the lexicon as our theory of semantics. Our ap- 
proach however is not limited to thematic roles. For an ex- 
ample of a more structured theory of semantics, see (Peterson 
and Billman, 1994). 



the arbitrator. This procedure would translate gram- 
matical relations to thematic relations and vice versa 
using other kinds of linguistic information such as ac- 
tive/passive voice, what roles go with a particular event, 
and so on. 

There are two problems with this approach. First, it 
is a procedural solution with well-known disadvantages 
over a comparable declarative solution. Second, during 
error recovery in incremental sentence comprehension, 
corresponding things will have to be undone in syntax 
and semantics. This is problematic if the only represen- 
tation of the correspondence is in the arbitration pro- 
cedures, since correspondence knowledge is not present 
in the representations that need to be manipulated by 
error recovery mechanisms. Also, the only kind of recov- 
ery possible is through reprocessing because there are 
no representations of intermediate decisions to repair or 
backtrack to. 

We propose an alternative to this in which the gap 
is bridged via intermediate representations. These in- 
termediate roles connect grammatical relations to the- 
matic roles, resulting in a declarative representation of 
the correspondence knowledge in the form of role hierar- 
chies. Now, with this mechanism, during error recovery, 
the processor can backtrack just the right amount and 
recover from the error by repairing the erroneous struc- 
ture rather than reprocessing from scratch. 

Role hierarchies that bridge the gap between syntac- 
tic phrases and thematic roles for example sentences are 
shown in Figure ij^ As the sentence progresses from 
that in (a) to the one in (b) or the one in (c), we can 
see the role structures being repaired to recover from lo- 
cal errors. It can also be seen that while both (b) and 
(c) involve repairs to role hierarchies, (c) also involves 
a reorganization of its syntactic structure to accommo- 
date its reduced relative structure. Thus (c) is more of 
a garden-path sentence than (b). 

The Uniform Representation 

In order for the single arbitration process to manipulate 
both syntactic and semantic types of knowledge both 
during early commitments and error recovery, we pro- 
pose a uniform representation of all types of knowledge. 
The elements of the uniform representation are units 
called nodes comprised of (a) part-whole links, (b) pre- 
conditions, (c) expectations, and (d) preference levels. 
The representations are to be read as (Fig. 2a) "the 
parts can be linked to the wholes when the precondi- 
tions are met and if so, the corresponding expectations 
can be generated at that point." A node represents all 
the knowledge about a syntactic or semantic category. 
Sample representations of syntactic and semantic knowl- 
edge in this general form are shown in parts (b) and (c) 



(a) The olllcers taught 




VP: "taught" 



(b) Jbe officers taught by the men.... 



Experiencer 



Passive-Subject 



Ageut 



Passive-Bv-NP 
[ 

Bv-NP 



Subject 

NP: "the officeL" VP: "taught" PP: "bj the men" 



Event: "give" 




Agent 

I 

Active-Subject Passive-Subject 



VP: "taught" PP: "at the academy" 
(c) The officers taught at the academy gave 



Figure 1: Role Hierarchies: From Phrases to Thematic 
Roles. 



^For simplicity, syntactic structures that are linked to the 
role hierarchies are not shown. 



of Figure 2. The NP node specifies the wholes S, VP, and 
PP of which it can be a part along with the preconditions 
and expectations therein. Similarly, the semantic node 
specifies knowledge of the thematic roles that go with 
a Teach event and the preconditions on the role fillers. 
The examples shown in Figure 2 are, however, simplified 
and incomplete. 

Intermediate roles such as Active- Subject, Non- Agent- 
Active-Subject, and Passive-By-NP are used to bridge 
the grammatical and thematic relations between the 
parts of a sentence. Their use in backtracking during 
error recovery can be seen by examining the transition 
from (a) to (b) and (a) to (c) in the examples in Figure 
1. 



Part-of: 

Preconditions: 
Expectations: 



Part-of: 

Preconditions: 
Expectations: 



NP: 




Part-of: S 




Preconditions: 


(precede NIL) 


Expectations: 


(follow VP) 


Part-of: VP 




Preconditions: 


(precede V) 


Expectations: 




Part-of: PP 




Preconditions: 


(precede PREP) 


Expectations: 





TEACH: 

Has-part: Agent 
Preconditions: (must-be animate) 
Expectations: 

Has-part: Experiencer 
Preconditions: (must-be animate) 

Expectations: 

Has-part: Location 
Preconditions: (must-be place) 

Expectations: 



B: a svntactic node 



C: a semantic node 



Figure 2: Uniform Representation of Linguistic Knowl- 
edge. 



As an alternative to uniform representations, if there 
are separate representations of different types of knowl- 
edge in different forms, it becomes hard to support incre- 
mental interaction in view of the constraint that the time 
course of actions taken by the processor must be psy- 
chologically real. Especially in the case of error recovery 
phenomena, it is elegant to have an arbitration process 
which processes all the kinds of knowledge (i.e., applies 
common operations on different kinds of knowledge) so 
that it can make corresponding changes to syntactic and 
semantic interpretations while recovering from an error. 



Processing with Uniform Representations 

The arbitration process has been implemented in a sen- 
tence understanding program called COMPERE. Its sin- 
gle arbitration process reads words in a left-to-right or- 
der, proposes ways of attaching them syntactically and 
semantically to previously built structures, follows the 
left-corner parsing algorithm mentioned above in pro- 
ducing the right sequence of decisions, and selects the 
most preferred attachments at each point by combining 
the preferences assigned to the alternative attachments 
proposed bottom-up with the grammatical, semantic, 
and structural types of top-down influence. COMPERE 
produces tree structures representing the syntactic struc- 
ture and thematic role assignment for a sentence. It 
is capable of representing and pursuing multiple inter- 
pretations in parallel when a conflict between syntactic 
and semantic preferences forces a delay in resolving an 
ambiguity. Further details of COMPERE'S processing 
mechanisms can be found in (Mahesh, 1993; Eiselt et al, 
1993). 

compere's arbitration process retains alternative 
interpretations that it did not select in the first place.^ 
If an error is detected at a later point (when there is 
no way to attach a new constituent, for instance), it re- 
covers from the error by switching over to one of the 
retained alternatives, performing any repairs to the syn- 
tactic and role structures (as shown in Fig. 1 for exam- 
ple) (Eiselt et al., 1993; Holbrook et al., 1992). The in- 
termediate roles in the role hierarchies help maintain the 
correspondence between the changes made to the syntac- 
tic and semantic structures. COMPERE maintains the 
correspondence information by uniformly connecting ev- 
ery syntactic node to its corresponding semantic node(s) 
(including the intermediate roles) and vice versa. 

In order to model the variety of behaviors, including 
the findings of Stowe (1991) on limited delayed decisions, 
the arbitration process must be enhanced by imposing 
architectural constraints, such as resource limits. The 
control structure of the process can then order the pref- 
erences based on their cost in terms of resources such 
as working memory. For instance, limited delayed de- 
cisions can be modeled by ordering semantic influence 
on par with syntactic preferences to begin with and by 
ignoring the semantic preferences when the processor 
has run out of resources. COMPERE'S separation of 
knowledge sources retains the functional independence 
between them which is necessary for the above enhance- 
ments. 

COMPERE can deal with a variety of (psy- 
cho)linguistically interesting constructs such as relative 
clause sentences, complements, and prepositional ad- 
juncts with both lexical and structural ambiguities (in- 
cluding the ones used in the examples above) . It can ana- 



For an account of the Conditional Retention theory of 
error recovery, see (Eiselt 1989). 



lyze their syntax and assign thematic roles. For example, 
it can show why sentence (1) is a garden-path sentence 
using grammatical expectations and early commitment; 
it can recover from the garden path to reinterpret the 
sentence; and it can use immediate semantic interaction 
to show why sentence (3) is not a garden path. The pro- 
gram has demonstrated the computational feasibility of 
the uniform representation and the arbitration process. 
However, its semantic competence is limited to thematic 
role assignment and at this time it does not have the 
knowledge to carry out discourse and reference process- 
ing. 

(3) The courses taught at the academy were 
very demanding. 

Related Work 

Though incremental interaction between syntax and se- 
mantics is not a new idea in computational modeling 
of human sentence comprehension, previous models that 
subscribed to such an interactive view have sacrificed 
the ability to apply syntactic and semantic knowledge 
independently of each other. While syntax-semantics in- 
teraction helps us explain a variety of psychological data, 
it is certainly not sufficient to account for the data on 
purely structural preferences, on deferred decisions with 
limited delays, and on error recovery phenomena (Eiselt 
et al., 1993). Below we briefly describe why some of the 
other models cannot explain the apparently incompati- 
ble data on modular effects (which functional indepen- 
dence can explain) as well as interactive effects (which 
incremental interaction can explain). 
Models with Integrated Representations: These 
models resort to an a priori integration of knowledge 
sources with a consequent loss of functional indepen- 
dence (e.g., Jurafsky, 1991). 

Categorial Grammars: Categorial grammars which 
establish a strong correspondence between syntactic and 
semantic categories (Steedman, 1989) also lack func- 
tional independence between syntax and semantics. 
NL-SOAR: Though this model, a contemporary of 
COMPERE, has a uniform representation and integrates 
multiple knowledge sources to resolve ambiguities, it can- 
not account for delayed decisions since it can only main- 
tain one interpretation at a time. In addition, its chunk- 
ing operations result in the integration of different types 
of knowledge with a consequent loss in functional inde- 
pendence (e.g., Lehman et al., 1991; Lewis, 1993). 
Connectionist Models: Connectionist models of sen- 
tence processing (e.g.. Waltz and Pollack, 1985), includ- 
ing the Competition Model of MacWhinney and Bates 
(Bates et al., 1991), also have uniform representations. 
However, they do not deal very well with the full syntac- 
tic complexity of natural language. It is yet to be seen if 
the simple computational mechanisms of activation and 
inhibition in a network can exercise enough control of 



Table 1: How COMPERE meets psychological con- 
straints. 



Constraint Features of COMPERE 

1. Increment ality Head-Signaled Left-Corner 

parsing algorithm. 

2. Early Commitment Grammatical, structural, and 

semantic top-down guidance 
to the parser. 

3. Delayed Decisions Arbitration mechanism and 

retention of alternatives. 

4. Error Recovery Arbitration mechanism, 

retention, uniform rep., 
declarative rep. of syntax- 
semantics correspondence 
through intermediate roles. 

5. Semantic Interaction Arbitration of independently 

proposed syntactic and 
semantic preferences. 

6. Functional Independence Separate representation of 

syntactic and semantic 
knowledge. 



processing to model the precise mechanisms of arbitra- 
tion, error recovery, and delayed decisions. 

Conclusion 

Table summarizes the features of COMPERE that help 
us satisfy the psychological constraints we started with. 

We have shown that in order to explain a variety of hu- 
man sentence processing behaviors, the sentence proces- 
sor must use a bottom-up strategy and yet accommo- 
date top-down influence from grammatical, semantic, 
discourse, and structural preferences. We presented a 
uniform representation of different kinds of knowledge in 
a common format but in separate units. Using this repre- 
sentation, we showed how we can arbitrate syntactic and 
semantic processes and account for a range of behaviors. 
We have demonstrated the computational feasibility of 
the model in the COMPERE program. We believe that 
the combination of an arbitration mechanism and a uni- 
form representation will take us a long way in modeling 
human sentence processing behavior. 
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